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IBSTFACT 

This publi^cation discusses a school evaluation aodel 
developed by the Center for Hev Schools (CHS) for use in evaluating 
Cleveland's Hew School. ' Section 1 describes three key principles that 
underlie the CHS aodel and discusses the rationale behind each of 
those principles. The three principles include (1) fa^cilitating 
coMunity participation in school evaluatiour (2) using a formative 
evaluation approach that encourages school improvement and (3) using 
^ eclectic multimethod approach to evaluation that employes J 
carefully trained and supervised participant observers. The fiml 
section presents, a step-by-step outline of the ifork plam for 
implementing CHS evaluation model at IJew School. (JG) 
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The CNS. model for 'evoluoting school pro^oms evolved K^m the exper fence of the 
•l^xecutTve As^ociotes in evolubting educotionol progroms in the lote 19605 , As part of our 
work in estoblis}>tng'^Aelro High School, we develo^d otjd implemented o very intensive ' 
resedrcf/progrpm on thotVchocrt*. As described in the proppsol, we ore now completing 
the onolysLs of thp Metro poto collected over o two yeor pe\iod. When we re,ceive<i.-requests 
to evoluote new schools/ w^a^knevv thot the Metro model wo^s too expensive. The CNS model 

/ ^ • 

for school evoluotidn, then, is built upon the reseorch principles of the Metro ev^luotion. 



modified to me^et the resources of schools, in o manner consistent with CNS ideas obout how 
CNS con be effective in promoting change. As we described in the propQSol, we see this 
model os on evolving one to be refined by octuot>e5cperience in terms of using it \\} the field, 
ond by.Vubjecting it, to criticism by ppai^mi^"^ reseorchers* The fol lowing description' * 
is of the mode] os we ore ppplying it toyfjew School in C!levelond ot the present time. The *^ 
first section describes Hie rotionole bejiind three key principles of the model ond the -second 
section describes the specific steps. Our work in usin^ the m<^ ot'New SchooP is progressing 
pn time ond very successfully to date. ' - ' ' , , ^ " 




Tig^^riaciples ^ 
- Three principles undertie^be^yaluotion modelt 




1 . * The SeliuoT Cilm n^ imity m Crv-^>^^^tof ^ _^ i 



of the community being jooked ot ore octive in determining what issues 

» .... 



' and processes should be concentrated on. Their participation in the 
inquiry process is also important as it reduces costs and increases the 
porti'cipantsi capabilities to perform continuous evaluation functions 
which they normally ore dependent on outsiders for« 

2. Evaluation as Formative Feedback, 
• Information about how and whysometKing happens is more useful than 

information that simply indicates that it did hoppen^'^lnformotion ' 
' should be gathered in a way that facilitates the communit/'s use of 
the data and as quickly as possibhe, . + " ^ . 



3, The Eclectic Multi-Method Apprcoch to Evaluation, 
* Sensitive and useful infomotion about a comrnunity is rT>ost Ukefy to 
result from an eclectic approach to methodologyjivhich focuses on the 
problems to be solved ami which gathei*s strength from a variety of ^ 
..Qjethodologies, ' 

1. School Community^ OS Co-Evoluotor. ' ' ' , 

\ 

' . r • 

Evaluation attenppt$ often folf victim to the obstacles of either too nauch inside perspective 
i or too much outside perspecti>^e. _Ti^e- internal evoluotor without external perspective often 
becomes too caught up in /be dajty reality of the organization to see the emergence of im- 

portont quesHon^^* ~ He U^ft^n ovefsociolized to the normj and mores of the community. His 

' " . - - _ ' '/ ' i ' 

^^^^^^ ^" . * > 

*Throughout this proposal the mole' pwioun is used for convertience. There is no assumption ' 
about whot sex the peopfe should octUi?ny be . ' , " — ' 



ottempts to evoloote often turn into sotidonty stotements reoffirming the) volues ond practices 

• 7 • 

of the community. ( 

The externol evoljjotor suffers from dther severe problems, > Typicolly, he comes for o 
short time, mokes some meisurements, leaves, ond writes o report. Whot he looks ot moy* ^. ' 
• derive from a superficiol underefanding of the school's goals or it moy derive from his own 
' ^j^ftori^ies obout what isiJmportont. The usuol reoctions of community members to this^kind 
\ • of evoluotion ofe that the importont issues were not inspected, thot the evoluotion misses 
the richnessbf whot goes on, and thot the evoluotion is not very useful. ^The^evo]^^ is 
se^n dt best os o complete outsider ond ot worst as o hostile judging invoder. The evol- 
_ uotor's obility to intefpto*.6v^iits ond his occess to information' ore limited by these per- 
tjons. - , - 

Most siihbol evoluotion in the post hos been done by external evoluotors, with the hope 
of ochieving on objectfve judgment of the progrom's worth. This hope has leen ottocked 
recently by severol social scientists. This recent c(fticism -suggests that th^^seorch for 
"objectivity" is elusive ot best, and that even the choice of working from on "external 
persp^cti^" is a value loaded act (egr Bruyn, 1966). * J - 




The Center for New Schools tries to establish its role somewhere in between. We believe 
that o fruitful evaluation must be conducted within the framework of the community's values.* 
Otherwise it is unlikely that the evoluotDl^ will be^oble to analyze the 'community on its^'OWrr"' 
^Igrms or give the community useful information. 

The error of oversociolizoti^must be ovorded. The evoluators, while working within 
* the value commitments of the community, must maintain on identity separate from direct 
community participants. They must stay somejwhot apart In-order to provide an outside per- 
spective thpt can help the community. . / -1 



Center for New Schools stoff, for exomple, hove had extensive experience participating^ 
in ond studying schools similor to^ New School, TTiese schools, while shoring mony of the 

•4 * I 

same goals and process commitments os New School moy hove hod different experiences. The 
evoluators use this richness of past expediences, then, to odd to the information that comes 
directly from New School, 

We pkin to establish this insider/outsider evaluation and to facilitate the dommunity's 
participation as co-evoluotor through the following steps, CNS yf\\\ mokeronlnitfdt field 
visit of several days to New School, During this visit we W^fHvseek out ^e stot^ and i/nstoted • ^ 
concerns of the community. We will conduct informo^^terviews of olf segments bf the 
community, listen to the concerns expressed in c9nversations between community members. 



observe the fc^rmol and informal activities of the participants, and generolly try to under- 
stond the spirii of the community, - - \ 

\ . ... ' . ^ ' ' \ 

V Only after; this inifiol ocquointonce process con we tentotively identify several subs^n- 



tive fcjci for the evaluation. While these issues will come directly from the concerns of ctm- 
munity members tfiemselves, they might include for exompie the following questions: Whot 
goals is New School successfully Achieving? Why js New School successful with these goals 
^ond less successful with o^thers? -A^hot problems existin the schod ? Why do these problems 
'exist? How ore various kirtd? of students (teochefs) dealing with counseling groupie (or ony 

other program element)? ^reJtudents lemming In New School? How Is Inforn)ation obout 

*** * ^ " • ' • 

the school c^ommunicoted witfrTfi the community? . 

• " . ' ' , 'i - 

Another way. the joint evoluotor relatlonStTip is promi^ted \% through the use of cJimmunity 

• - \ • " 

members os information gatherers wherever (jpsslble, Membeis'hove o wealth of background 
experience in the^community which facilitates their occ^s It? iirformatlon. They hove rap- 



-5- ^' r 

• • ■ :■ ( ' 

port with other members which on outsider would hove to spend much time building. 'Similorly, 
they hove kn6wledge pf the'longuoge ond concrete experiences of the community which serves 

OS a backdrbp to whof people soy ond do. ' 

■ . - ' " ^'f • ' - 

Therefore^ GNS Involves community members in.the entire informotion collection process. 

We consult with members In the design of questionn'oires ond interview schedules. In most 

coses, community members odminister these instruments. The use of participant obsjervation 

similarly guorontees the octive porticipotion of the community in'^the colleiSion of informotion. 

W^^ train ond support the co-evoluotors so.t^ot they will be sensitive to the information^ 

they gather and con ovoid the dongers of oversociolizotion. We co-ordinote ond supervise 

^hese people-offering help where needed ond constontly providing perspectives to enrich the 

dato gophering, Finolly, we monitor the d^to being gothered ond provide gui^once for changed 

foc^^here necessory. 

2. . Evoluotlon OS Formotive Feedbock , _ * _ ~ 

The usuol opproaches to evoluation of educotionol settings hove concentr|i(ed on whot 
is called ^/summotive evoluation" (Bloom, Hostings, ond Moddous, 1972). In this model, 
certoiff outpyt voriobles ore specified ond me.osured ot the end of some period. The evoluo- 
ion ottempts tcr "sum up" the bccomptishmpnts or lock of occomplishment. 

SeveroT .deficiencies con be^rwpted with this method. The informotion often comes to!8 
lote to help the people in the setting to improve^'their programs. Furthermore', the method 
of gathering information is not designed to^rQvide o^sistance« ' Sunw retA'tro v a I u e t i ofn — ^' 
ottempts to tell to whot degree vorious goals were ochieved, but it roF^W gives ony clues 
obout why or how those levels were achieved. Also it does not wsuall)^ give any indicotion 



of how different gtoups of people reocted to the progrorti. 




people p 



/ 



"Formative evoluotion" is on oltemotive opprooch^ Its .intent, in oddition to providip^ 
Info/motion obout effects, is to generate insights, helpful 4o the orgonizotlon^^^^fmg the 
process of evoluotion rother than just ot the end. Also it is designed-t9/^ther o kind of 
detoiled informotion thot is linked close enough to the doily reolify^ it becomes 6f immediote 
use to people in the setting. V ' 

CNS promotes formotive evoluotion in severol woys, A primory hope is thot fhe evolu- 
otion will stimi/lote on orggnizotionol self-sfudy process,thot continues beyond the formol 
involvement of^CNS« 

Mony of the processes described in^fhe previous ag^tion olso support formotive evoluotrc^n. 

The co-evoluotor relotionsjjip^encourages the,(2^munlty*5 willingness ond eogemess to use 

the information gyathered.-.The initial ifelp visit stimulates community members to increase 

their critical thinking about therfoctivities and reactions. Individuals in schools like New 

School" ord^irf^jjoljy introspective about their organization long before fprmal evaluation starts; 

« 

the visit of on outside agency like CNS encourages this indigenous process, gives tools and 
assistance to aid in the methods, and gives o frofnework to organize the process and moke it 
comprehensive. 

The use of comm^^^ty members to formulate questions ^nd to gother information guarantees 

ckjoops esto^mhed^ the community. Furthetmore, the 



that there will be immediate feedbo 

\ 

experience community members gain in the piiocess \^ill be useful for the evolu^ation th9t con- 

tinues after th^ end of the CNS fdrmol involvement. The use of participant observation as 

principql method of studysjikewise guarantees immediate feedljasl^ to the community^. Suc- 

^. \ . \ ^ * • 

cessful porticjpant Observation requires constant interaction with community members. 

. '"x ' ' , • '■' 

CNS wiU insti^jte formative evoluo^tion in two direct ways. During the period of in- 
volvement, we will run two feedback workshops with the community. Based on emeiging 



insights, we will ottempt to stimulote thought ond oction designed to help members movd — 
toword thejr gopjs/ Finolly, CNS will wrife its reports with the oim e^f^erving the school . 
in its further growth ond development. * . 

3, Eclectic Al[\ultr"Method. Approach to Evoluotion. 
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Most reseorch ond evoluotion in educotion hos be'en based on on onoly1'i>col 'model 
borrowed from the physicol sciences* In this opprobch, the evoluotof decides opriori whot 
the relevo^t voriables will be, designs objective methods of ossessment which purposely ex- 
elude olh subjectivity, opplies these methods ovoidlag ony kind of relotionship with the 
people involveo, onflKtRert\withdraws to moke his conclusions. Both theorists ond proctitionsers 
hove begyn to ottgck these methods os missing the most importont ospects'bf on educotionol 
pr<^rom. As on oltemotive wey oppose methods long used in onthropology 
field studies. (Scott, 1965.) 

One cor^loint obout the usui^l methods is thot thesy ore too orfifici 
(1968) ond his os^o^jot^s ^^^^ showi^ thot hurrion beings.^ offectg/by thejr surroundings. 
Much behovlor is shope;! by^he-^pbysitflf^'ond sotfai context p{ octivity. Methods such 
OS questionr^oires, tests, ond interviews, give limited kinds of infontiotion becouse they creote 
their own context not necessarily related to the progroms under study, Borker proposes natu- 

V 

rolistic observotion where whot people soy ond do is'observed in its reLoJ cohtext^ith the 
evoluotor being ds unobttusive os possible. Educotors ?uch os Phillip Jacks<^,(1968) ohd 
Louis Smith (Smith & Geoffrey, 1968) hove shown that this opptooch is expeciolly importont 
in educotional settings. - The usuol designs have misled. the complexity ond interrelofionships 
thot become opporent when one observes these settifigs;wholisticolly. 

\ Another. complaint otout troditiohol evoluotioa is that the seorch^icJy^^ forces 
the evoluotor to neglect much thot is importont in under^tonding whot hoppens in an orgoni- 



peo|5le ottoch to gools ond ev^ntfe oncf thcyfe^'ng they hove 

/ \ { ( ^ 

obout them ore often excluded in studies where oil the vorioWei o 



zotion. Philosophies of sociol science^uch os Bruyn (1966) or^ GJoser & Strauss (1967) 
ossert'thot the meanings thot pe^le ottoch to gools ond ev^n)/onJ the, 

or^mported from the out- 
side. Methods must be uied which oilow the community's meoniiigs to guide the explorotion. 
Anthfopologlcol ai^d sociologicol field studies (sometim^^^edV'qu the best 

modesi for tli^ie methods. 

Applicotion of these quoJitotive methods to educotionol evoluotion hos been rore in the 
r. The troditional qudntitotive method of evoluotionJtave been very populor becouse they 




ore pimple to design, ^lotively inexpensive to opfity, onfl give ofluringly simple results. 
Those, hdwever, who wont evoluotion which does not do violence to the doily reolity ond 
which is'usefyl to those octuolly inyolve^Mn the orgonizotioh ,ore not sotisfied with conven- 
tionol Evoluotion opprooch. They Wont the kind of evoluotion thot pays heed to the integrity 
of their orgdnizGtions, 

CNS ho^§ye+oped"on eclectic multi-method opproach to evoluotion "oTnew school f^ro- 
groms. The opproqch is eclectic in thot the first question thot is osked is what is the purpos 
•of the evoluotion. Methods hove then been designed to g«+^ier information thot is consi 
with thot purpose. Estoblished 4^sedrch desigfns techniques Qre used4^ insure strenj 
reliability bf*the opprooches used, ond not os^he predicotor for those dppcoach^^ 
we hoVe developed on ,oppnDach thot relies on several rperhods of inf 
multi-method opprooch WovJdes greoh^r richness of doto, 
.between the opprooches. Our plon for New^choc 



views (formol ond informol), ond porticif 
tion is {presented in the next section. Sj 



t isjlie most unusuol of the severoLrapproochesy >ve will describe 



to 




UO-- 



jrticipont obsep^otion is seen as key ond sfnce 



our rgtfotiale for using it 




-9- 
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below m more ^etolL The ftiethodologlcol feotyre of most importance to^ur opprpocn, 
however, is the combinotion of seyero 

Pdrticipant observotion serves os o key backdrop for the whole evaluation of tf^pro- 
grqm. We pJon to use a half time port 
•period. Researchers in the past hove found 
approach to naturalistic stufly and, to I 




School d'urihg^eNftvofuation 



observation is the sfiogle best 
[yatfon stays in' tune' to the mean- 




ings of the community being studied, 

. The participdnf observer combines the 'tnsider/outsider perspectjAfesJ He joins a com- 
munity and attempts to undergo experiences much as parti cipants^.wou Id. He becomes port 
of the formal and informal activities of the schooK 1;^ wotchei what hapj3ens, he listens 
to what people say, he becomes trusted "and exfierienced enough so that participants feel ^ 
comfortable sharing their inner reactions witli him, and he asks questions whrch sMmufate 
these reactions. At the some time;,' He guards -ago'inst becoming foo^ much on insider. Throug^v^ 
special techniques^ he guards ogoirtst over-sociolizaiion. Similarly, he mokes sure that het 
doeslliot adopt the perspective oH&iy particular group in the school . He doily recdnds his 
observations ond reoctions in a w6y thot fodilitcites their use in [ater.dnalysis. (The metho^ 
dology jaf [>articipant otservotion is too complex to describe fully hi^;e; fot more detoFIs se6 . 
McCol I & Simmons, 1969). ' . ' ^ . 

CNS has hod experience in thifif past using participant observation in evol-Cioting* alternative , 
schools. We will Iroin/and supervise ^the pdrticipdhf observer in th^ use of these methods. In 
addition we will mop/tor the information being gathered to Kelp interpret infon^iation gathered 

by other methods, to 'guide future data gqthering, to reflect emerging copcerrts of the,com- 

/ - " ' , ' ' . ^ ' • ' . 

munity, oncJ to provide formalize feedback ^o the, community. The participant observer>by , 



11 



ill 



z_ 



tbe speciol nature -of his insider-oul'sider r<ile ^onstontfy provides formotive feedback in the 
questions osks end by his very, presence whicly^ulates ^e community to be''mt^^ctiv^?t * 




In these Woys porticipofit observotion serves purposes discussed in the previ<^iS-two ^tions.# 

mrk Plon for-EvolAJofJon-of JSleV Scbool 

^ 

Activities to be corried put. 
^ 1 . Defihitibn of issues. 1st Site Visit. 

Tosk: Define the mojor issued for the^evCluotion. NS Evqiucrtion Committeeiwi 

orronge o series of meetings with stoff, studfenh^ jond odmiijiistrators to iden- 
^tify key^ issues. CMS will conduct interviews ond some informql observation * 
to olso identffy jssues. The '^cluotion Committee ond CNS Will meet^in 

■ ^ two sessions to compile the finol Hst.. ^ 

2. Agreet^en^^ on Evalu^^op- Procedures. Isjt Site Visit. ^ 

Tosk: Decide on whotmethods wiJI be used t6 collect^ oniTyze ond disscminote ^ 
* informotio'n . Meth<5ds of inquiry sbouKd be developed thot ore consistent 
^ « D ' with re^eorch principles, ond vJhich meet locoi needs in ^terms of cost, 

^ >hysicol logUticfs^ ortci other ccWtcuQts. CNS, cte described in the ration- 
ale, hos definite judgements on what methods ore appropriote bas^d on our 
post e)5perl^ncfe. Decisions. ore mgde l^^CN^<jnd. th^ Evoluotion Committee. 

• 3. Pref)orgtion of lntervie>V ond Gjrudstionnoire Schedules.'' 

Tosk: Prepare instrUment^^Tiiot w<ll be uSed to collect ond anqjy^e th€*k)formotion. 
, CNS, conferrin^fremiervtly wit>) the Evq[uation Committee, devefops the 
instrument ond proeeapres that wiM te'used to select respondents, ond' • 
onolyze informoHon. ■ ^ . 

4. Initiol Tromiog of Interviewers ond Porticipont Qbserver, 2r\<i Site Visit. 

^ T^k: Troin'pdrticipont observed ond progrom participants who wlU^^ col-le<f^i'ng 
Jnformotipn. (Porticipont observer oppllcants dr^ screened by the^l^luo- 
tmn Committee and formolly selected by CN5.) Troining of interviewers 
to ocbTfiinister inter^^iew includes o pilot- inter\?iew adj|pmlstered/Dy eocf 
interviewer.'* This process: ol$o serves jas a pr€j-test,(or*the \nmxMi€^^ The 
* porticipont observer is trained o^ site.during^thejgiijr-^ifer^;^^ 

. I.' J I * - ^ ' . . * '-'^ • * . ► ' . 

5. Vollow-up tfoirving of Porticipont Observer ond Troining of R^oluot4<}h Committee ^ 
^ iri^lnitial Arialysis PrbceduVes^^tj^ite Visit (2 days). 



Tosk; F6II0W up- on-site with tKer Porticipont Observer in terms of problems 
be i$ fdcing. Train evoJ4jotion commiltee in procedures to onolyze'in- 

Tormotion collected from 5. . . : - » ^ 

♦ 

/6. First Collection of Informqtion ond Porjficipont ObserverJ^egins Work. ' 

/.""^^Tdff^orkshop. 4th Site Visit. • * 

Tosk: Prepare works'hop based on pertinent inforrrxati^ for stoff to donsidfer"* 
Informationrto be presented comes from' both^onolysis of interviews dind 
from observotions of the Porticipqnt Observer; 



8i Second Dofd Collection. 




Tect^informatjon that will indicote chor^es'in participants* skills, 
'perspectives obout the progrom*ond their poMicipation in it^ The 
second ^to^ col lection is bi^Ut o round the samte or'similor instruments 
used itt the first. ' * c ■ \ 

* • :^ ^ 

9. Program Porticipont Anolysis of Dato; . ' * j ^ 

Tosk: Ahalyze informatron ftonTsecond "rfata cbilectfoi%or>d of differerfces • 
between the two f5eriods of time. Report forworded to CNS. / * 

10. Preporotion of Sec.Odd Staff, JA^orkshop ond oflfeport ta School Boord. 

Tosk: Prepare o workshop that relote* to the i^ues foced by^stafC^ond program 
^participants, based on information collected. ^Prepare fornto^-report f dr 
Board of Educotion. * * \ ' v 

11. Present Workshop. , . , 

12. Provide contlnuirig support and coordination during process o^^yoluation. 

13; Secorid Yeor follow Up. 

Tosk^"Hrovide program participants with support as they tolce Oft olmost fuM^^ 
responsibility for continuous self-evaluation, based on the training ond 
procedures of the first year. As described in the Rationale, CNS-worksi 
to insure that the schootsusing H^is model develop their own copobiljties 
rather than remain dependent. \ ' 
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